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PREFACE 


This  research  supports  the  Air  Combat  Tactics  and  Training  Thrust  of  the 
Air  Force  Human  Resources  Laboratory.  By  describing  an  approach  to  provide 
the  sensory  and  perceptual  data  needed  in  the  aircrew  training  device  (ATD) 
design  process,  the  research  directly  supports  the  Engagement  Simulation 
Technology  Subthrust  of  the  AFHRL,  Operations  Training  Division.  The  work  was 
accomplished  under  contract  by  the  Boeing  Aerospace  Company.  This  paper 
consists  of  material  that  was  written  and  submitted  by  the  contractor,  but  has 
been  reorganized  and  edited  by  the  Government. 

This  research  program  was  conceptualized,  initiated,  and  managed  by  Dr. 

Ken  Boff,  under  the  former  Advanced  Systems  Division  of  the  Air  Force  Human 
Resources  Laboratory.  The  program  at  Boeing  Aerospace  Company  was  managed  by 
Mr.  Wolfe  Hebenstreit.  The  principal  research  activity  was  accomplished  by 
Mr.  Richard  Farrell  and  Dr.  Richard  Barker. 

Although  the  research  program  described  herein  was  terminated,  a 
reorganized  approach  to  achieve  the  original  project  goals  was  initiated  by 
the  Aerospace  Medical  Research  Laboratory,  with  the  support  of  the  Air  Force 
Human  Resources  Laboratory.  That  program,  entitled  Integrated  Perceptual 
Information  for  Designers  (IPID),  is  managed  by  Dr.  Ken  Boff. 
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1.0  INTRODUCTION 


The  goal  of  the  Integrated  Cuing  Requirements  (ICR)  Study  was  to 
consolidate  and  synthesize  existing  human  sensory/perceptual  data,  principles 
and  models  in  a  manner  which  would  make  this  information  readily  accessible 
and  useful  to  the  community  of  aircrew  training  device  (ATD)  design  engineers. 

There  exists  an  extensive  body  of  research  literature  on  human  perception 
which  could  potentially  be  of  value  in  the  specification,  design,  and 
evaluation  of  aircrew  training  devices.  The  data  in  this  domain  are 
distributed  among  numerous  different  publications  and  are  written  in  the 
specialized  terminology  of  perceptual  psychology.  Consequently,  this 
information  is  not  generally  accessible  to  ATD  engineers.  The  goal  of  the  ICR 
Study  was  to  extract  and  consolidate  the  relevant  data  into  an  accessible 
format  and  to  provide,  where  feasible,  a  synthesis  of  the  literature  which 
included  recomnendations  relevant  to  equipment  design.  The  intended  output  of 
this  activity  was  (a)  an  ICR  Data  Base  containing  the  available  sensory/per¬ 
ceptual  data  in  a  form  useful  for  specification  and  design  purposes  and  (b)  an 
ICR  Users  Guide  to  facilitate  access  to  the  data  by  the  ATD  engineer. 

Volume  I  of  this  technical  paper  presents  the  results  of  an  Independent 
Feasibility  Analysis  (Section  2.0)  and  a  subsequent  Demonstration  Study 
(Section  3.0),  the  purpose  of  which  was  to  develop,  test,  and  refine  an 
approach  to  the  ICR  Study  objectives  discussed  above.  Appendices  A  and  B, 
respectively,  present  the  Literature  Search  and  Review  Plan  and  the  Data  Base 
and  Users  Guide  Development  Plan  for  the  achievement  of  these  objectives. 

These  activities  constituted  Phase  I  of  the  effort.  The  project  was 
structured  such  that  no  decision  on  full-scale  implementation  of  the  approach 
(Phase  II)  would  be  made  until  an  evaluation  of  Phase  I  had  been  completed. 
Volume  II  of  this  technical  paper  contains  the  Phase  I  Demonstration  Data  Base 
and  Users  Guide.  Figure  1  diagrams  the  planned  sequence  for  Phases  I  and  II 
of  the  program. 
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2.0  INDEPENDENT  FEASIBILITY  ANALYSIS 


2.1  Objectives 

This  analysis  was  intended  to  improve  the  study  approach  by  soliciting 
critiques,  recommendations,  and  fresh  insights  from  recognized  experts  in  the 
areas  of  perception,  training  psychology  and  simulation  engineering.  The 
analysis  was  to  provide  an  objective  assessment  of  the  feasibility  of  the 
basic  study  assumptions  and  the  overall  approach  as  well  as  assessing  the 
likelihood  of  successfully  achieving  the  study  objectives. 

2.2  Approach 

Three  experts  representing  the  appropriate  areas  of  expertise  initially 
conducted  indeoendent,  structured  reviews  of  the  study  plan.  This  was 
followed  by  a  team  review  at  the  contractor's  facility  to  produce  recommended 
changes  and  feasibility  assessments.  An  overview  of  the  assessment  is 
presented  in  Figure  2. 

2.3  Detailed  3lan 

2.3.1  Selection  and  Approval  of  Experts 

Three  experts  were  selected  and  approved  by  the  Air  Force  Contract  Monitor 
(AFCM)  for  accomplishing  this  analysis.  Personnel  selected  and  their  areas  of 
expertise  were  as  follows: 

Dr  Hershel  W.  Leibowitz:  Perception 

Evan  Pugh  Professor  of  Psychology 
Pennsylvania  State  University 

Dr  Robert  L.  Howe:  Computer  Systems 

Department  of  Aerospace  Engineering 
University  of  Michigan 
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Figure  2  Independent  Feasibility  Assessment  Plan 


s 


Dr  Robert  R.  Mack ie : 

President,  Human  Factors  Research,  Inc. 


Training 


2.3.2  Development  of  Structured  Review  Form 

A  structured  review  form  was  developed  to  enhance  objectivity  and  control 
the  scope  of  the  reviews.  The  experts  were  encouraged  to  add  additional 
comments,  criticisms,  and  recommendations.  The  review  form  was  intended  to 
serve  as  a  checklist  of  critica1  areas  in  the  planned  approach  and  suggest 
criteria  for  the  assessment  of  effectiveness  and  feasibility.  An  outline  of 
the  form  is  presented  in  Table  1.  It  was  designed  to  help  assure  that  the 
details  of  the  intended  approaches  to  the  individual  program  elements,  as  well 
as  the  overall  goals  of  the  program,  were  properly  assessed. 

2.3.3  Briefing 

A  briefing  of  the  review  team  was  held  at  the  contractor's  facility  to 
establish  a  solid  base  for  the  subsequent  reviews.  Copies  of  the  Statement  of 
Work,  relevant  plans,  and  the  structured  review  form  were  sent  to  the 
reviewers  prior  to  the  briefing  to  provide  an  initial  orientation.  This 
briefing  providec  in-depth  descriptions  of  the  study  objectives  and 
assumptions  as  well  as  the  study  approach  as  defined  by  the  literature  search 
plan  and  the  pre' iminary  comprehensive  study  plan.  The  briefing  included  a 
discussion  of  the  feasibility  analysis  objectives,  its  importance  in  the 
overall  study  approach,  and  detailed  instructions  for  conduct  of  the  reviews. 


TABLE  1 

REVIEWER'S  CHECKLIST 


A.  Study  Assumptions 

B.  Literature  Search  Plan 

1)  Search  Terms 

2)  Literature  Sources 

3)  Screening  Criteria 

4)  Prioritization 

5)  Evaluation  Criteria 

6)  Sunmary  Formats 

C.  Data  Base  Development  Plan 

1)  Preliminary  Topic  List 

2)  Approach  to  Identification  of  Topics 

a.  ATD  Design  Parameter  Categories 

b.  Training  Cue  Categories 

c.  Perceptual  Categories 

3)  Data  Integration  Process 

a.  Combined  Pilots 

b.  Principles  and  Models 

4)  Methods  of  Presentation 

a.  Entry  Techniques 

b.  Recommendations 

c.  Tutorial  Material,  Analyses,  etc. 

d.  Extracted  and  Formatted  Data 

5)  Organization  of  Users  Guide/Data  Base 

D.  Demonstration  Study 

1)  Specific  Problem  Areas 

a.  Data  Availability 

b.  Impact  on  ATD  Design 

2)  Simulated  Design  Exercises 

3)  Impact  on  ATD  Spec ification  and  Evaluation 

E.  Overall  Assessment 

1)  Feasibility 

2)  Problem  Areas 

3)  Potential  for  Approach  Improvement 


10 


2.3.4  Independent  Reviews 


The  initial  reviews  were  conducted  independently  by  each  expert  at  his  own 
facility.  A  period  of  approximately  two  weeks  was  used  for  this  process. 

Each  of  the  experts  prepared  materials  for  the  team  review. 

2.3.5  Team  Review 

After  the  individual  reviews  had  been  completed,  the  review  team  met  at  the 
contractor's  facility  to  compare  and  collate  responses  to  the  structured 
review  form,  general  comments,  and  recommendations.  These  results  were 
reviewed  by  contractor  personnel  with  the  review  team  to  establish  the 
desirability  and  feasibility  of  incorporating  the  recommended  changes  into  the 
study  approach. 

2.3.6  Presentation  of  Results 

The  results  of  the  team  review  were  jointly  presented  to  the  AFCM  by  the 
contractor  and  independent  experts  at  a  meeting  held  at  the  contractor's 
facility.  This  presentation  included  the  following  program. 

o  Probability  of  success  for  the  planned  program. 

o  Recommended  changes. 

o  Probability  of  success  with  recommended  changes. 

2.3.7  Definition  of  Study  Modifications 

After  review  of  the  recommended  changes,  contractor  personnel  and  the  AFCM 
selected  specific  changes  based  on  feasibility  and  likelihood  of  payoff. 

These  changes  were  incorporated  in  the  Demonstration  Study. 

2.4  Structured  Review  Form 


Appendix  C  contains  the  complete  text  of  the  review  form  for  the 
Independent  Feasibility  Analysis.  A  summary  of  the  results  of  the  analysis  is 
contained  in  Section  2.5. 

2.5  Results 

The  consensus  of  the  team  was  that  the  ICR  Study,  as  conceived,  is  a  major 
undertaking  that  would  make  a  much  needed  contribution  to  the  design  of  air¬ 
crew  training  devices  and  that  this  contribution  would  have  a  positive 
cost/benefits  impact.  Within  this  general  finding,  the  team  had  the  following 
specific  comments  and  questions: 

It  was  pointed  out  that  the  ICR  project  objectives  involve  identifying 
both  the  data  needed  to  support  ATD  design  and  the  data  available  in  the 
literature  to  meet  these  requirements.  Through  the  comparison  of  these  two 
sets  of  data,  gaps  in  the  available  data  should  be  identified  in  a  systematic 
fashion,  and  these  gaps  must  be  prioritized  in  terms  of  their  need  for 
inclusion  in  future  research  programs. 

The  availability  of  a  broad  base  of  established  cue  requirements  upon 
which  to  draw  during  the  development  of  the  Data  Base  and  Users  Guide  was 
questioned.  Such  information,  if  available,  would  be  of  significant  use  to 
the  ICR  program.  Its  unavailability  would  result  in  some  lack  of  specificity 
that  might  detract  from  the  utility  of  the  product.  There  was  general 
agreement  that  cue  requirements  information  specific  to  air-to-air, 
air-to-surface,  and  nap-of-the-earth  flying  was  far  from  complete. 

Concern  was  expressed  over  the  ability  of  the  contractor  to  select, 
review,  and  summarize  the  pertinent  literature  without  introduction  of 
unintentional  bias  due  to  the  limited  number  of  reviewers  employed.  In 
response,  it  was  noted  that  the  contractor's  in-house  team  had  been  expanded 
by  the  addition  of  two  subcontractors:  one  in  the  area  of  audition  and  the 
other  in  vestibular  perception  of  force  and  motion. 

The  issue  was  raised  of  the  relationship  of  the  ICR  study  to  the  overall 
simulator  procurement  process.  While  the  influence  of  improvements  in 
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hardware  capability  on  simulator  design  was  not  the  topic  of  the  ICR  study,  it 
was  deemed  essential  to  keep  this  capability  in  mind  when  establishing  ICR 
data  requirements  and  when  developing  the  data  base  materials.  It  was 
anticipated  that  the  Air  Force  technical  support  team  could  contribute 
significantly  in  this  area. 

Concern  was  expressed  that  a  Demonstration  Study  topic  be  chosen  which 
would  be  sufficiently  specific  to  permit  development  of  an  adequate  sample 
Data  Base  and  Users  Guide  on  which  basis  the  Phase  I  program  would  be 
evaluated. 
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3.0  DEMONSTRATION  STUDY 


3.1  Introduction 

The  requirements  imposed  on  an  ATD  vary  as  a  function  of  the  flight  task  to  he 
trained.  An  ATD  suitable  for  the  landing  task  is  not  necessarily  adequate  for 
air-to-air  combat.  Although  training  mission  analysis  was  not  within  the 
scope  of  the  ICR  Study,  it  did  place  emphasis  on  data  relevant  to  the  design 
of  ATDs  for  training  the  most  combat -critical  portions  of  flight.  Hence,  data 
relevant  to  nap-of-the-earth  flight  or  air-to-air  combat  would  receive  more 
attention  in  the  ICR  Study  than  data  relevant  to  takeoff  and  landing. 

The  ICR  documentation  is  divided  into  two  parts:  the  Data  Base  and  the  Users 
Gu ide . 

The  Data  Base  contains  the  extracted  research  data,  organized  for  easy 
access  and  summarized  in  terms  of  recommendations  or  implications  for  ATD 
design.  The  format  facilitates  expansion  beyond  the  currently  planned 
sections  if  additional  topic  areas  are  identified  as  useful. 

The  Users  Guide  provides  a  means  of  accessing  and  utilizing  the  Data 
Base.  Because  access  is  so  important  in  a  document  intended  for 
engineering  use,  different  Data  Base  entry  methods  are  provided  to  meet 
the  needs  of  individual  users.  These  include  tables  of  contents,  an  index 
and  several  types  of  design-or iented  analyses.  (In  addition  to  the  entry 
methods  in  the  Users  Guide,  access  to  related  topics  is  provided  within 
the  Data  Base  by  cross  references  to  other  Data  Base  sections.) 

Preparation  of  the  Data  Base  and  Users  Guide  followed  an  Air 
Force-approved  Preliminary  Comprehensive  Study  Plan  developed  early  in  the 
program.  The  major  steps  called  for  in  these  plans  are  summarized  in  Section 
3.3  below. 

As  with  the  Data  Base,  these  plans  were  developed  in  a  preliminary  form 
and  reviewed  by  independent  outside  consultants,  whose  comments  and 
suggestions  were  used  to  improve  the  product. 


The  primary  methods  of  access  in  the  sample  User's  Guide  were  a  surmtary 
table  of  contents,  a  detailed  table  of  contents  listing  every  subsection  in 
the  Data  8ase,  and  an  index.  A  brief  description  was  also  included  of  other 
methods  of  access  planned  for  inclusion  during  the  proposed  Phase  II  portion 
of  ICR.  These  would  provide  access  in  terms  of  specific  hardware  features  or 
in  terms  of  the  training  mission  of  an  ATD  (for  example,  low-level  flight  or 
air-to-air  combat). 

3.2  Sample  Data  Base  and  Users  Guide 

The  preparation  of  the  sample  Data  Base  followed  a  series  of  steps 
designed  to  ensure  that  the  data  were  maximally  useful  to  the  ATD  engineer  and 
were  scientifically  valid.  These  steps  included: 

1)  Identification  of  the  data  topics  pertinent  to  ATD  specification, 
design,  or  evaluation.  This  step  utilized  the  experience  of  ATD 
engineers  and  of  scientists  involved  in  human  perception  who  had 
participated  in  ATD  specification  or  evaluation. 

2)  Selection,  from  the  list  developed  in  Step  1,  of  two  sample  data 

topics  for  incorporation  into  the  sample  Data  Base.  These  were 

(a)  judgment  of  surface  orientation  and  shape,  and  (b)  visual 
induction  of  self-motion. 

3)  Collection  and  review  of  research  literature  on  these  sample  topics. 

4)  Extraction  from  this  literature  of  the  research  data  relevant  to  ATD 

design . 

5)  Formatting  of  these  data  for  easy  interpretation  by  ATD  engineers. 

6)  Summarization  of  these  data  into  preliminary  recommendations 
applicable  to  ATD  specification,  design  or  evaluation. 

7)  Review  of  the  preliminary  recommendations  by  outside  expert 
consultants. 


8)  Incorporation  of  the  consultants'  comments,  changes  and  suggestions 
into  the  material. 

9)  Preparation  and  delivery  of  the  sample  Data  Base  material  for 
evaluation  by  a  representati ve  group  of  ATD  engineers. 

A  sample  Users  Guide  was  prepared  to  provide  access  to  specific  topics  in  the 
sample  Data  Base. 

3.3  Evaluation  Procedure 

An  independent  group  of  ATD  design  engineers  was  selected  by  the  Air  Force 
to  serve  as  reviewers  of  the  Phase  I  product.  These  reviewers  were  asked  to 
respond  to  a  total  of  eight  specific  questions  covering  three  general 
evaluation  topics.  These  topics  were  (a)  the  accessibility  of  the  data  in  the 
Data  Base,  (b)  the  formatting  and  presentation  of  the  data,  and  (c)  the 
usefulness  of  the  data.  General  comments  were  also  requested. 

To  assist  the  reviewers  in  making  their  evaluation,  each  was  provided  with 
a  series  of  hypothetical  design  problems.  By  attempting  to  solve  these  design 
problems,  the  reviewer  obtained  a  better  understanding  of  what  data  the  Data 
Base  contained  and  how  adequately  data  on  a  specific  topic  could  be  located 
and  interpreted. 

The  text  of  the  user's  evaluation  form  appears  in  Appendix  D. 

3.4  Results 

For  each  of  the  evaluation  questions,  most  of  the  reviewers  provided  a  general 
response,  such  as  "good"  or  "poor",  and  a  number  of  specific  responses  in  the 
form  of  comments  about  particularly  good  or  poor  features.  Responses  to  six 
of  the  eight  questions  are  summarized  in  Figures  3  through  8.  (Not  all 
reviewers  responded  to  all  questions,  so  the  number  of  reviewers  represented 
in  these  figures  varies  from  9  to  18.)  Questions  to  which  fewer  than  nine 
reviewers  responded  are  not  included  on  this  summary. 
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Each  figure  contains  (a)  the  specific  question  from  the  evaluation  form  to 
which  the  results  apply,  (b)  a  histogram  of  reviewer  "good "/"poor"  responses 
to  the  question,  (c)  a  statement  of  the  percentage  of  these  responses  that 
were  favorable,  and  (d)  an  anecdotal  sampling  of  the  specific  comments  that 
the  question  elicited. 

The  histograms  in  these  figures  serve  to  illustrate  that  the  responses 
were  generally  favorable.  The  reviewers  rated  the  Demonstration  Study  sample 
ICR  material  as  well  presented,  easily  accessible,  and  potentially  useful. 

Specific  corrments  included  both  positive  and  negative  evaluations  of 
particular  aspects  of  the  sample  material  and  suggestions  for  refinements. 

All  of  these  comments  contributed  to  the  identification  of  features  which 
should  be  retained  or  improved.  For  example,  the  results  indicated  in-depth 
design  examples  should  be  added  to  make  it  apparent  to  engineering-oriented 
users  how  to  apply  the  data  to  specific  simulator  design  problems. 
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FIGURE  3 


A3  WHICH  METHOD! S)  OF  ACCESS  ARE  MOST  EFFECTIVE? 
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The  underlined  Recommendations  were  distracting. 
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With  a  comprehensive  access  system  that  doesn't  take  an  hour  to  wade  through,  someone 


4.0  CONCLUSIONS 


1 


The  results,  though  based  on  a  limited  sample  of  data,  were  consistent 
with  the  general  conclusion  that  the  objectives  and  associated  products  of  the 
ICR  study  are: 

a.  Feasible 

b.  Useful  to  ATD  design  engineers. 

However,  an  assessment  by  the  Air  Force  management  team  also  led  to  the 
unequivocal  conclusion  that  a  significant  restructuring  of  the  planned 
approach  would  be  required  in  order  to  provide  the  depth  of  data  coverage  and 
synthesis  needed  to  achieve  the  study  objectives.  In  particular,  it  was 
concluded  that  it  was  unlikely  that  any  one  contractor  could  provide  the 
required  depth  of  coverage  and  associated  expertise  within  the  Government's 
cost  constraints.  The  existing  contractual  effort  was  therefore  terminated. 

A  restructured  approach  to  provide  direct  access  to  the  required  expertise  was 
designed  and  implemented  by  the  Air  Force  Aerospace  Medical  Research 
Laboratory  in  cooperation  with  the  Air  Force  Human  Resources  Laboratory. 


APPENDIX  A 


LITERATURE  SEARCH  AND  REVIEW  PLAN 

1.0  Figure  A-l  diagrams  the  literature  search  and  review  plan.  The  literature 
search  and  review  will  begin  with  a  statement  of  the  required  data  described 
in  the  Data  Base  Development  Plan.  These  requirements  will  be  prioritized  and 
used  to  direct  a  search  of  the  literature.  When  promising  literature  is 
obtained,  it  will  be  screened,  summarized,  and  evaluated  for  inclusion  in  the 
ICR  data  base.  Access  to  and  control  of  the  literature  obtained  will  be 
achieved  by  means  of  a  computerized  bibliography.  Data  covering  sensory/per¬ 
ceptual  topic  areas  specified  in  the  preliminary  and  final  comprehensive  stuoy 
plans  will  be  extracted  from  the  literature  that  meets  the  evaluation  criteria 
and  will  be  formatted  and  integrated  into  the  ICR  data  base. 

2.0  Evaluation  of  potentially  useful  literature  will  occur  at  several  points 
during  the  search  and  review.  To  reduce  the  chance  of  missing  useful  litera¬ 
ture,  the  threshold  for  obtaining  a  copy  of  a  potentially  useful  article  will 
be  kept  very  low.  The  resulting  literature  must  then  pass  several  increasingly 
critical  evaluations  before  reaching  the  point  where  a  significant  amount  of 
contractor  time  is  spent  in  extracting  and  formatting  data. 

3.0  To  ensure  the  literature  search  and  review  is  directed  toward  the  most 
important  data  topics  and  the  most  relevant  data  sources,  priorities  will  be 
established  throughout  the  program.  The  data  base  content  requ i*-ements  will 
be  prioritized  to  establish  which  topics  will  receive  the  largest  initial 
emphasis.  During  the  literature  screening  process,  articles  will  be 
prioritized  for  surmiar ization  and  evaluation  on  the  basis  of  their  probable 
contribution  to  the  data  base. 

4.0  Rapid  feedback  from  the  results  of  the  literature  search  and  re.  v  will 
be  essential  to  ensure  particular  topic  areas  are  not  being  overworked  or 
underworked.  This  feedback  will  be  obtained  by  continual  review  of  the 
outputs  from  the  literature  search  and  review  as  articles  are  summarized  and 
evaluated  for  content  and  as  data  are  extracted  and  formatted  for 
incorporation  into  the  ICR  data  base.  By  comparing  these  outputs  with  the 


data  base  content  requirements,  it  will  be  possible  to  determine  what  changes 
in  emphasis  are  needed  in  the  process  of  searching  for  and  identifying 
additional  literature.  This  evaluation  of  the  output  from  the  literature 
search  and  review  will  also  help  in  establishing  the  final  data  base  content 
requirements. 

5.0  Most  topic  areas  will  be  the  subject  of  several  publications.  Knowledge 
of  these  similar  articles  is  important  during  the  evaluation  process  because 
they  indicate  the  uniqueness  and  hence  the  importance  of  a  particular  study  on 
the  topic  area,  and  they  provide  comparative  results.  It  is  also  essential  to 
have  these  groupings  of  articles  identified  prior  to  formatting  the  data  from 
an  article  because  it  will  generally  be  necessary  to  place  the  data  in  a  con¬ 
sistent  format,  or  even  to  place  data  from  several  studies  on  a  single  figure. 
Many  of  the  several  articles  on  a  single  topic  area  will  be  identified  through 
the  reference  lists  in  the  more  recent  articles,  some  will  be  identified 
through  subject  indexing  within  the  b ib liography,  and  others  will  be  identified 
by  program  personnel  as  they  search  for  literature  on  a  particular  topic. 

6.0  The  specific  sensory /perceptual  areas  in  which  data  will  be  extracted, 
formatted,  and  integrated  into  the  ICR  data  base  will  be  controlled  by  the 
Data  Base  Development  Plan. 

7.0  The  bibliography,  literature  summaries,  and  extracted  and  formatted  data 
will  be  reported  in  the  form  of  a  Technical  Literature  Review.  As  this 
material  is  integrated  into  the  ICR  data  base,  the  resulting  data  base  will  be 
submitted  to  the  Air  Force. 

8.0  Literature  Evaluation 

8.1  Each  article  that  appears  promising  for  incorporation  in  the  ICR  data 
base  will  be  formally  evaluated.  The  purpose  of  this  evaluation  is  to 
establish  how  safe  it  is  to  make  use  of  the  results  and  conclusions  reported 
in  the  article  in  the  ICR  data  base.  Forms  will  be  used  to  provide  a  record 
of  each  evaluation  and  to  ensure  the  evaluator  can  quickly  and  efficiently 
consider  all  the  evaluation  topics.  To  reduce  the  time  lost  on  irrelevant 
topics,  the  evaluator  will  have  the  option  of  not  entering  a  formal  rating  for 
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EVALUATE 

CRITERIA 


Figure  A-1  Literature  Search  and  Review  Fiow 


evaluation  topics  that  have  little  relevance  to  the  usefulness  of  a  particular 
article. 

3.2  Evaluation  criteria  will  vary  with  the  type  of  article  involved.  For  a 
report  of  an  experiment,  the  evaluation  will  consider  the  reliability  and 
validity  of  the  experiment  and  the  experimental  results,  and  the  correctness 
and  degree  of  certainty  in  the  conclusions  drawn  from  these  results.  For  an 
analytical  article,  the  concern  will  be  with  the  validity  of  the  author’s 
initial  assumptions,  analysis,  and  conclusions.  Tutorial  and  review  articles 
will  be  treated  like  analytical  articles  except  that  the  concern  will  be  with 
the  completeness  and  representativeness  of  the  literature  covered  rather  than 
with  the  starting  assumptions. 

8.3  Evaluators  for  a  particular  article  will  be  selected  on  the  basis  of 
their  expertise  in  the  technical  area  covered  by  the  article.  The  number  of 
evaluators  will  depend  on  the  importance  of  the  information  contained  and  on 
the  potential  controversy  involved  in  applying  this  information  to  simulator 
design.  In  most  cases,  a  single  evaluator  in  addition  to  the  individual  who 
sumnarized  the  article  will  be  sufficient. 

9.0  Data  Extraction  and  Formatting 

9.1  For  data  evaluated  as  useful  contributions  to  the  ICR  data  base,  the 
relevant  information  contained  in  each  article  will  be  extracted.  The  summary 
of  the  article  will  serve  as  a  starting  point,  but  more  details  will  be 
necessary,  particularly  in  the  area  of  experimental  conditions  and  results. 

9.2  The  specific  information  to  be  formatted  for  each  article  or  group  of 
articles  will  be  that  required  to  integrate  this  material  into  the  ICR  data 
base.  This  will  include  the  experimental  results  obtained,  usually  in  a 
quantitative,  easily  assimilated  graphic  form,  plus  whatever  background 
details  are  necessary  to  interpret  and  use  these  data.  This  will  generally  be 
the  same  type  of  information  summarized  earlier,  with  enough  detail  for  the 
data  user  to  understand  how  the  data  were  obtained  and  how  much  reliance  can 
be  placed  in  the  data. 
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9.3  This  information  will  be  formatted  for  each  access  and  interpretation. 

In  some  cases,  it  will  be  possible  to  simply  copy  the  original  author's 
presentation  of  his  results  and  to  combine  these  with  a  sumnary  of  the 
experimental  conditions.  In  many  other  cases,  the  original  figures  will  not 
be  adequate  for  purposes  of  the  ICR  study  and  changes  will  be  required.  These 
might  be  simple  changes  in  scaling  to  allow  comparisons  of  results  across 
different  studies.  Taking  a  simple  example,  in  some  studies  in  which  scene 
illumination  level  is  an  independent  variable,  the  results  are  reported  in 
terms  of  scene  luminance  and  in  others  they  are  reported  in  terms  of  retinal 
illuminance.  During  the  formatting  process,  these  different  units  would  be 
converted  to  a  single  unit  that  would  allow  the  integration  of  results  across 
the  studies. 

9.4  In  some  cases,  data  formatting  will  involve  extensive  manipulation  of  the 
experimental  results.  This  might  involve  reinterpreting  the  test  data  in 
terms  of  different  parameters  of  more  use  to  the  aircrew  training  device 
designer  than  the  parameters  used  by  the  original  authors.  In  a  few  cases, 
reinterpretation  may  be  required  because  the  original  authors  missed  some  of 
the  implications  of  their  results.  This  occurred,  for  example,  in  a  recent 
review  of  experimental  data  on  viewing  distance  requirements  for  television 
displays.  In  one  study,  the  original  authors  presented  their  test  data  on 
target  detection  in  terms  of  the  angular  subtense  of  the  television  monitor, 
but  these  data  were  both  more  consistent  and  more  useful  to  the  review  when 
they  were  recomputed  and  sunmarized  in  terms  of  resolution  elements  across  the 
target  at  detection. 
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APPENDIX  B 


DATA  BASE  AND  USERS  GUIDE  DEVELOPMENT  PLAN 


1.0  Approach 

At  the  simplest  level,  data  base  development  consists  of  taking  the 
information  available  in  each  of  many  topic  areas,  analyzing  this  information 
to  determine  the  implications  for  ATD  and  human  factors  engineering,  and  then 
presenting  these  implications  and  the  supporting  information  in  the  ICR  data 
base.  Carrying  out  this  activity  will  require  numerous  interactive  steps,  as 
can  be  seen  in  the  sumnary  of  the  planned  data  base  development  illustrated  in 
Figure  B-l.  The  resulting  products  are  summarized  in  Table  B-l. 
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TABLE  B-l.  Data  Base  and  User's  Guide  Development  Products 

PRODUCT  CONTENT 

ICR  USER'S  GUIDE  PURPOSE  OF  USER'S  GUIDE  AND  DATA  BASE 

ENTRY  TECHNIQUES  (USER  ORIENTED) 

TABLES  OF  CONTENTS 

( SUMMARY  LEVEL  &  DETAILED) 

INDEX 

POTENTIAL  CUE  CONFLICTS 
POTENTIAL  CUES  FOR  PARTICULAR  TASKS 
HARDWARE  FEATURE-RELATED  ANALYSIS  AND 
CHECKLISTS 

MISSION-RELATED  ANALYSES 
DESIGN  EXAMPLES  ILLUSTRATING  DATA  BASE  USE 
TUTORIALS  AND  BACKGROUND  ON  TECHNICAL  AREAS 
GLOSSARY,  DEFINITIONS,  ABBREVIATIONS, 

ACRONYMS  AND  UNITS 

GUIDE  TO  PSYCHOPHYSICAL  METHODOLOGY  AND 
STATISTICS 

ICR  DATA  BASE  RECOMMENDATIONS,  SUGGESTIONS  FOR  DESIGN 

ANALYSES  AND  DISCUSSIONS 
EXTRACTED,  FORMATTED,  AND  INTEGRATED  DATA 
EVALUATION  OF  DATA  ADEQUACY 
IMPLICATIONS  OF  INDIVIDUAL  DIFFERENCES 
CROSS  REFERENCES  TO  OTHER  TOPICS 

SUPPORTING  DATA:  LITERATURE  SUMMARIES  AND  EVALUATIONS* 

*NOTE  -  UNLIKE  THE  USER'S  GUIDE  AND  DATA 
BASE,  THE  LITERATURE  SUMMARIES  AND 
REVIEWS  WILL  NOT  BE  DELIVERED  IN  A  FORM 
SUITABLE  FOR  GENERAL  DISTRIBUTION. 


1.  ATD-ORIENTED 


2.  HUMAN  FACTORS 

ENGINEERING  ORIENTED 
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Figure  B-1  Data  Base  and  Users  Guide  Development  Flow 


2.0  Data  Selection  and  Integration 


The  heart  of  the  data  base  development  process  is  the  integration  of  the  data 
covering  a  particular  topic  area  into  the  form  in  which  it  will  appear  in  the 
data  base.  This  ’’s  a  process  that  involves,  for  each  topic  area,  a  series  of 
steps.  In  many  cases,  these  steps  are  performed  iteratively. 

2.1  A  promising  set  of  data  from  an  experiment  or  group  of  experiments  that 
appears  to  be  relevant  to  the  topic  area  is  identified. 

2.2  The  report  of  each  experiment  is  reviewed  to  determine  that  the 
experiment  was  properly  performed  and  the  results  are  valid.  Factors 
considered  include  the  test  hardware  and  procedure,  the  test  subjects  and 
their  training,  the  experimental  design,  and  the  data  analysis. 

2.3  The  results  of  other  experiments  are  reviewed  to  determine  if  they 
confirm  or  contradict  the  data. 

2.4  The  data  are  integrated  with  data  from  other  comparable  experiments  if 
these  are  available  and  if  the  additional  data  are  needed  in  the  data  base. 

2.5  The  data  are  summarized  in  an  easy-to-interpret  graphic  format. 

2.6  The  data  and  the  experiment(s)  that  led  to  these  data  are  described. 

This  description  covers  what  was  done  in  the  experiment  and  the  implications 
and  limitations  of  the  resulting  data. 

2.7  Design  recommendations  summarizing  the  implications  of  the  data  for 
hardware  and  system  design  are  prepared. 

The  data  integration  activity  will  benefit  from  numerous  techniques  and 
sources.  For  example,  when  several  experiments  are  available  that  indicate 
the  contribution  of  a  particular  variable,  these  can  be  considered 
simultaneously  by  computing  the  proportion  of  the  variance  in  each  experiment 
that  was  accounted  for  by  the  variable.  Review  articles  and  texts  summarizing 
a  particular  topic  area  will  provide  guidance  in  understanding  the  area  and  in 
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integrating  the  available  data.  Perceptual  principles  and  models,  eithe» 
previously  published  or  provided  within  the  government-furnished 
sensory/perceptual  data,  will  provide  a  tentative  framework  within  which  the 
available  data  can  be  integrated  and  summarized.  It  will  be  essential  r;o 
assess  the  usefulness  of  this  integration  by  comparison  with  the  data  base 
content  requirements.  For  example,  a  description  of  the  principles  underlying 
the  perception  of  motion  in  depth  may  be  useful  as  a  means  of  integrating  a 
disparate  set  of  test  data  and  may  be  useful  to  the  scientific  community  as  a 
source  of  hypotheses  to  be  tested  experimentally.  However,  it  belongs  in  the 
ICR  data  base  only  if  it  contributes  to  an  understanding  of  how  best  to 
achieve  ATD  or  human  engineering  design  goals.  As  the  sections  of  the  data 
base  are  completed,  they  will  be  reviewed  by  potential  users  who  will  attempt 
to  use  them  to  solve  hypothetical  design  problems.  The  results  of  these 
reviews  will  then  be  used  to  revise  the  data  base. 

3.0  Entry  Technique  Development 

Because  the  information  in  the  data  base  will  be  used  extensively  only  if  it 
can  be  easily  located,  major  emphasis  will  be  placed  on  the  development  of 
access  methods.  With  the  exception  of  cross  references  from  one  technical 
area  to  the  other  within  the  data  base,  these  access  methods  will  be  contained 
in  the  Users  Guide. 

The  access  methods  provided  will  be  organized  around  the  needs  and  interests 
of  the  potential  users.  For  example,  the  emphasis  in  the  index  will  be  on  the 
terminology  of  engineering  rather  than  perceptual  psychology.  This  goal  will 
be  achieved  by  extensive  interaction  with  potential  users  before  and  during 
the  preparation  of  the  access  methods,  and  later  by  revising  these  methods 
based  on  reviews  by  potential  users. 

4.0  Data  Base  Organization 

The  ICR  data  base  will  be  structured  to  make  it  easily  accessible  to  the 
primary  users,  ATD  designers  and  human  factors  engineers.  To  achieve  this 
goal,  the  technical  material  in  the  data  base  will  be  organized  into  sections 
and  subsections  that  contain  the  fundamental  sensory/perceptual  information 
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used  in  the  specification  of  design  requirements.  Each  of  the  design  problem 
areas  requires  information  from  more  than  one  section  of  the  data  base. 
Conversely,  each  section  of  the  data  base  will  apply  to  several  design  problem 
areas.  The  Users  Guide  ill  direct  the  user  to  the  sections  applirable  to  his 
design  problem. 

Within  these  design-oriented  topics,  the  information  of  importance  tn  a 
particular  user  will  vary  according  to  the  intended  function  of  the  hardware 
or  system  being  designed.  For  example,  an  ATD  intended  for  training  an 
individual  to  detect,  identify  and  assess,  and  respond  to  the  intended  action 
of  another  aircraft  in  a  tactical  combat  scenario  must  meet  :nore  stringent 
image  resolution  criteria  than  one  intended  for  trainin'  an  individual  to  land 
an  aircraft.  These  types  of  task-related  differences  in  ATD  design 
requirements  will  be  treated  in  the  ICR  Users  Guide,  where  the  available 
knowledge  on  the  cues  that  may  be  involved  in  performing  various  aircrew  tasks 
will  be  discussed.  This  discussion  will  make  the  ICR  user  aware  of  the  cues 
that  should  be  provided  when  training  a  particular  type  of  performance  and 
will  direct  him  to  the  sections  in  the  ICR  data  base  where  they  are  discussed 
in  more  detail.  Within  the  data  base,  the  impact  on  design  requirements  of 
these  variations  in  cue  requirements  for  different  tasks  will  be  discussed  and 
the  appropriate  data  base  material  (recommendations,  implications  for  design, 
supporting  research  data,  etc.)  will  be  presented  for  each  major  task 
application. 

Within  particular  topic  areas,  perceptual  principles  and  models  will  serve 
several  functions.  In  some  cases,  they  will  provide  a  basis  for  design 
recommendations  and  approaches.  For  example,  control  system  models  of  the 
human  operator  provide  insight  into  permissible  simulator  transport  delays. 

In  other  cases,  perceptual  principles  and  models  will  provide  the  framework 
around  which  the  available  research  data  on  a  particular  topic  can  be 
organized.  As  an  example,  this  approach  will  be  useful  in  treating  topics 
such  as  spatial  orientation,  where  many  features  of  the  operator's  environment 
contribute  nonlinearly  to  the  perceived  c.  :entation. 
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5.0  Required  Data  Base  Content 

The  goal  of  the  ICR  program  is  to  provide  perceptual  information  to  improve 
equipment  specification  and  design.  Therefore,  the  specific  topics  to  be 
incorporated  into  the  ICR  data  base  will  be  selected  to  represent  the  user's 
needs.  This  list  will  undergo  extensive  evolution  during  the  data  base 
development,  utilizing  inputs  from  analyses  of  both  equipment  designers  and 
users.  Principles  and  models  in  the  areas  of  sensation  and  perception  will 
help  interpret  the  needs  of  the  users  and  organize  the  data  in  each  of  these 
areas . 

The  following  criteria  will  be  used  in  modifying  the  data  base  content 
requirements  list  and  to  help  in  establishing  which  topics  have  the  highest 
priority  for  consideration  during  the  ICR  program.  In  order  of  preference, 
the  greatest  emphasis  will  be  placed  on  the  following: 

5.1  Perceptual  principles  and  data  that  are  directly  applicable  to  widely 
recognized  design  problems  or  to  simulation  techniques  as  applied  to  design 
f  eatures . 

5.2  Perceptual  principles  and  data  of  general  value  to  equipment  design. 

5.3  Perceptual  principles  and  data  related  to  the  basic  processes  involved  in 
user  activities. 

5.4  All  other  types  of  data  or  information  deemed  useful  or  applicable  to 
equipment  design. 

Within  each  of  these  categories,  the  available  research  data  will  be  utilized 
based  on  the  technical  quality  of  the  work  and  the  extent  to  which  the 
experimental  design  and  subject  population  allows  generalization  to  equipment 
design  requirements. 

The  methods  of  presenting  information  in  the  data  base  will  be  designed  to 
serve  users  with  differing  interests  and  backgrounds.  Whenever  feasible,  the 


36 


presentation  of  each  topic  area  will  start  with  recommendations  or 
implications  for  design.  These  will  be  supported  by  sumnar ies  of  data  from 
relevant  research  literature,  formatted  for  easy  access  by  the  user.  The 
analysis  by  which  the  design  recommendations  were  derived  from  the  research 
data  will  be  included.  In  this  way,  the  user  who  is  severely  time  Imp  iter  car 
take  advantage  of  the  recormiendation  and  implications-f  or-design  sta terpen:  '■ 
developed  during  preparation  of  the  data  base,  while  those  users  who  have  a 
need  tG  do  so  can  trace  to  the  supporting  data. 

The  specific  methods  of  presenting  material  in  tire  data  base  will  be  trie 
following: 

Recommendations  and  Implications  for  Design  -  These  will  represent  •re 
best  summary  available  of  a  technical  topic  area  as  it  applies  directly  i- 
the  specification  or  design  of  a  particular  ATD  feature  intended  to 
perform  some  particular  training  task.  Care  will  be  taken  to  make  these 
recommendations  and  imp! ications-for-des ign  statements  clear,  unamt i  - 
and  operationally  oriented  so  they  will  lead  directly  to  useful 
procurement  specifications. 

Analysis  and  Discussion  -  This  material  will  expand  on  the  recommend¬ 
ations  and  implications-f or-design  statements  and  show  how  they  result 
from  the  research  data  summarized  in  the  data  base. 

Research  Data  -  The  research  data  relevant  to  the  technical  topic  ico1 
will  be  summarized  in  a  form  that  facilitates  comparisons  between 
different  research  studies  and  an  assessment  of  how  much  empirical  supp' 
is  available  for  the  design  recommendation  of  their  validity  and 
reliability  and  a  summary  of  the  test  conditions  sufficient  to  aiiow  the 
data  base  user  to  assess  whether  the  results  apply  to  a  particular  A'I'O 
situation.  Bibliographic  citations  will  be  included  for  the  benefit 
those  users  who  wish  to  consult  the  original  sources. 

Tu tor i a  1 s  -  When  necessary  to  ensure  the  data  base  user  will  understand 
the  implications  of  the  data  and  analysis  in  a  particular  topic  area, 
background  tutorial  material  will  be  provided.  This  material  wll  be 


presented  at  a  sunmary  level  and  will  include  references  to  publications 
where  a  more  complete  treatment  can  be  found. 

Frequent  use  will  be  made  in  the  data  base  of  cross  references  to  relevant 
material  in  other  sections.  These  will  serve  the  same  function  as  the  entrv 
techniques  provided  in  the  Users  Guide,  in  that  they  will  help  make  the  use'- 
aware  of  the  several  topic  areas  that  should  be  considered  to  solve  a 
particular  design  problem. 

6.0  Users  Guide 

The  primary  purpose  of  the  ICR  Users  Guide  is  to  provide  means  for  easily  and 
rapidly  accessing  and  applying  the  material  in  the  ICR  data  base.  To  achieve 
this  goal,  several  entry  techniques  will  be  provided  in  the  guide  so  that 
users  with  different  interests,  backgrounds,  and  levels  of  experience  with  the 
data  base  can  determine  what  information  within  the  data  base  is  potentially 
relevant  and  can  quickly  and  easily  locate  this  information. 

Among  the  entry  techniques  that  will  be  incorporated  into  the  User's  Guide  are 
the  following: 

Table  of  Contents  -  Two  tables  of  contents  will  be  included  to  provide 
access  to  information  about  a  particular  topic.  One  will  be  at  a  summary 
level  and  is  intended  for  the  individual  desiring  to  quickly  determine  the 
overall  scope  and  organization  of  the  data  base.  The  other  will  list 
essentially  every  subsection  heading  within  the  data  base  and  will  provide 
a  means  for  locating  material  on  specific  topics.  The  summary  table  of 
contents  will  not  exceed  two  pages  while  the  detailed  version  will  be  six 
to  perhaps  ten  pages  long. 

Index  -  An  index  will  be  included  to  provide  access  to  information  about 
a  particular  subject.  To  make  the  index  more  useful,  it  will  contain  both 
key  words  and  phrases  used  in  the  data  base  text  and  terms  descriptive  of 
the  content  but  not  actually  used  there. 
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Design  Features  Analysis  -  This  analysis  will  provide  access  to  the  da;  . 
base  in  terms  of  specific  hardware  or  software  features.  Examples  of 
design  features  include  display  field  of  view,  CRT  raster  line  visibility, 
and  spatial  distribution  of  computer  generated  texture  elements.  Within 
each  feature,  a  reference  to  a  location  in  the  data  base  will  include  a 
brief  statement  of  why  the  material  in  that  section  is  relevant. 

Design  Features  Checklist  -  This  list  will  serve  to  review  for  the  user 
the  various  topics  that  should  be  considered  in  the  specification  or 
design  of  an  ATO  intended  for  a  particular  purpose.  Included  with  each 
topic  area  in  the  checklist  will  be  an  indication  of  the  data  base  sectic: 
or  sections  where  it  is  covered.  It  is  anticipated  the  checklist  items 
will  be  phrased  as  questions.  For  example,  the  checklist  for  an  ATD 
intended  to  be  used  in  training  aircrew  personnel  to  recognize  and  respond 
to  other  aircraft  in  a  tactical  air  combat  situation  might  include  these 
entries : 

-  Are  the  visual  display  resolution  and  contrast  adequate? 

-  Can  the  displayed  other  aircraft  be  made  perceptually  equivalent  to  the 
operational  situation? 

-  Is  the  visual  display  field  of  view  adequate? 

-  Will  the  visual  display  collimation  impose  realistic  requirements  on  the 
visual  accommodation  of  aircrew  personnel  during  search? 

Mission-Related  Design  Feature  Analyses  -  This  analysis  will  provide 
access  to  the  data  base  by  the  design  features  required  to  simulate  a 
particular  type  of  mission.  For  example,  a  simulator  designed  to  train 
pilots  in  landings  and  takeoffs  will  probably  not  be  adequate  to  train  for 
nap-of-the-earth  flight.  A  simulator  designed  to  train  air-to-air  combat 
will  require  higher  resolution  than  one  designed  to  train  for 
nap-of-the-earth  flight  but  would  not  require  as  much  scene  content. 
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Potential  Cue  Conflicts  -  This  section  will  describe  the  cue  conf  '  ori  s 
that  may  result  from  a  particular  hardware/sof tware  implementation  of  3" 
ATD  and  indicate  where  these  are  treated  in  the  data  base. 

Potential  Cues  for  Particular  Tasks  -  Each  aircrew  task  has  a  unique  s>- 
of  cues  that  must  be  considered  during  ATD  spec  if icat ion  and  design  '> 
perceptual  fidelity  is  to  be  achieved  in  the  training  situation.  A 1  thou q 
the  specific  cues  involved  in  each  task  are  not  fully  established,  a 
considerable  body  of  literature  exists  that  provides  useful  guidance. 

This  literature  will  be  briefly  summarized  in  the  Users  Guide  to  aid  ir e 
ATD  specifier/designer  in  deciding  which  cues  to  incorporate. 

In  addition  to  these  entry  techniques,  the  Users  Guide  will  incorporate  "the'- 
material  to  aid  individuals  involved  in  ATD  specification  or  design.  These 
include: 

Role  of  the  ICR  Users  Guide  and  Data  Base  -  This  section  will  summarize 
the  content,  organization,  and  structure  of  the  Users  Guide  and  date  b  a  st¬ 
and  will  describe  how  these  can  be  accessed  and  used  to  improve  ATD 
specification  and  design. 

Guide  to  Psychophysical  and  Statistical  Methodology  -  This  section  ” 
intended  for  individuals  involved  in  testing  or  otherwise  evaluation 
ATDs.  It  will  summarize  some  of  the  approaches  to  such  evaluations, 
describe  seme  of  the  numerous  potential  pitfalls,  and  suggest  potential 
sources  of  experienced  guidance  and  assistance. 

Design  Examples  -  To  make  the  accessing  and  application  of  the  materia 
in  the  date  base  more  apparent,  this  section  will  contain  examples  of 
typical  ATD  design  problems  and  illustrate  how  they  would  be  improved 
through  use  of  material  from  specific  sections  of  the  data  base. 

Tutorials  -  This  section  will  contain  background  technical  material  to 
assist  data  base  users  in  understanding  and  interpreting  the  data 
presented  ;n  the  data  base. 
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APPENDIX  C 


STRUCTURED  REVIEW  FORM 


1.0  Study  Assumptions 

The  ICRS  approach  is  based  on  the  four  assumptions  listed  below.  Please 
conment,  as  appropriate,  on  the  potential  validity  and  the  importance  to  stud 
success  for  each  of  these  assumptions. 

1.1  Information  germane  to  the  questions  of  cue  utilization  and  the  influent: 
of  cues  on  control  inputs  exists  in  the  basic  perceptual  literature. 

1.2  Perceptual  information  germane  to  the  questions  of  cue  utilization  and 
the  influence  of  cues  on  control  inputs  will  be  useful  for  defining  the 
engineering  requirements  for  training  simulators. 

1.3  The  source  and  content  of  cues  are  a  function  of  the  aircraft  system, 
mission  and  individual  variables  relating  to  the  pilot  and  his  previous 
training.  The  scope  of  the  present  study  does  not  permit  a  thorough 
investigation  of  all  possible  combinations  and  permutations  of  these 
parameters,  nor  are  there  good  reasons  to  believe  that  this  would  be  a 
reasonable  approach  to  pursue.  Instead,  for  the  present  investigation,  we  a> 
assuming  that  (a)  there  are  some  aspects  of  cues  that  are  constant  across  ear 
of  these  parameters  and  that  (b)  detailed  information  relating  to  these  con¬ 
stants  can  be  useful  for  the  design  and  evaluation  for  retrofit  design  of  All 

1.4  ATDs  designed  using  ICR  data  will  be  more  effective  training  devices. 

2.0  Literature  Search  and  Review  Plan 

2.1  General  Approach 

(a)  Are  the  planned  activities  feasible? 


(b)  What  is  the  likelihood  for  success? 


(c)  Where  are  the  potential  problem  areas? 

(d)  What  can  be  done  to  improve  the  effort? 

2.2  Literature  Sources 

Please  indicate  any  suggestions  for  additions  to  the  planned  list  of 
sources. 

2.3  Screening  and  Prioritization  Criteria 

Several  levels  of  screening  are  planned  to  select  the  most  useful  and 
relevant  material.  Do  you  have  suggestions  for  modifications  or  additions  to 
the  criteria  planned  for  this  process? 

2.4  Evaluation  Criteria 

Evaluation  forms  are  planned  for  rating  the  most  relevant  literature.  Do 
you  have  suggestions  for  modifications  or  additions  to  the  criteria  listed? 

2.5  Summary  Form 

Reviewers  will  be  expected  to  summarize  individual  articles  using  a 
standardized  format.  Do  you  have  suggestions  for  modifications  or  additions 
to  the  planned  Report  Summary  Form? 

3.0  Data  Base  Development  Plan 

The  development  of  the  Data  Base  and  User's  Guide  is  certainly  the  most 
critical  step  in  the  ICRS.  The  following  items  are  considered  to  be 
especially  important  for  effective  performance  in  this  effort.  Your  comments 
concerning  the  feasibility  of  the  current  plans  and  suggestions  for 
improvement  are  requested. 

3.1  Preliminary  Topic  List 
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3.2  Approach  to  Identification  of  Topics 


(a)  ATD  Design  Parameter  Categories 

(b)  Training  Cue  Categories 

(c)  Perceptual  Categories 

3.3  Data  Integration  Process 

(a)  Entry  Techniques 

(b)  Design  Recommendations 

(c)  Tutorial  Material  and  Analyses 

(d)  Extracted  and  Formatted  Data 

3.4  Organization  of  Users  Guide/Data  Base 
4.0  Demonstration  Study  Plan 

4.1  Specific  Problem  Areas 

The  general  area  of  spatial/temporal  mismatch  has  been  selected  for  the 
demonstration  study  topic.  This  total  area,  however,  is  too  broad  for  the 
planned  level  of  effort.  Specific  items  will  be  considered  based  on  the 
availability  of  useful  information  and  the  significance  of  the  expected  impart 
on  simulator  design.  Suggestions  for  or  against  specific  items  to  be 
considered  will  be  helpful. 

4.2  Simulated  Design  Exercises 

This  step  is  intended  to  enhance  the  contribution  of  the  user  population 
to  the  development  of  the  organization  and  format  of  the  User's  Guide  and  Data 


Base.  Do  you  consider  this  approach  to  be  a  reasonabe  one?  Can  you  suqgest 
alternate  or  additional  efforts  to  maximize  user  contributions  to  the  study 
effort? 

4.3  Impact  on  ATD  Specifications  and  Evaluation 

The  preliminary  Users  Guide  and  Data  Base  developed  in  the  demonstration 
study  is  expected  to  provide  the  basis  for  demonstrating  +he  validity  of  the 
study  assumptions  and  the  feasibility  of  the  planned  approach.  The  intent  is 
to  use  a  range  of  expertise  to  identify  examples  of  significant  contributions 
of  these  materials  to  simulator  design.  Can  you  suggest  a  better  approach  for- 
demonstrating  the  potential  benefits  of  the  planned  study? 

5.0  Overall  Assessment 

In  addition  to  the  individual  items  noted  above,  your  assessment  of  the 
overall  study  approach  is  desired.  Summary  statements  are  needed  in  the 
following  areas: 

(a)  Feasibility  of  the  overall  study  approach. 

(b)  Likelihood  of  successfully  achieving  the  study  objectives. 

(c)  Identification  of  potential  problem  areas. 

(d)  Suggestions  for  improving  the  likelihood  of  success. 

Comments  on  the  overall  structure  and  scheduling  of  the  program  should  be 
included  as  appropriate. 
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APPENDIX  D 


USERS  EVALUATION  FORM 

The  ICR  Study  will  integrate  the  data  on  human  perception  applicable  to  the 
design  of  Aircrew  Training  Devices  (ATDs)  in  a  manner  that  will  make  these 
data  helpful  to  engineers  charged  with  the  specification,  design  or  evaluation 
of  such  equipment.  The  output  from  ICR  will  be  a  set  of  documents  containing 
the  integrated  data  and  providing  a  means  for  easy  access. 

A  small  sample  of  the  ICR  documentation  has  been  prepared  and  is  currently 
undergoing  review.  By  involving  the  ATD  engineer  in  this  review,  the  final 
product  will  benefit  from  the  opinion  and  experience  of  the  ultimate  user. 
Because  of  your  position  and  expertise,  you  have  been  requested  to  participate 
in  this  review. 

As  you  review  this  material,  keep  in  mind  that  what  you  see  in  the  document 
before  you  is  but  one  small  portion  of  the  final  total  output  of  this  study. 

As  a  result,  it  covers  only  a  small  sample,  perhaps  five  percent  at  most,  of 
the  large  body  of  relevant  and  potentially  useful  data. 

Our  primary  concern  is  to  achieve  a  document  which  will  have  maximum  utility 
to  the  engineer  involved  with  ATDs.  Thus,  we  will  be  asking  you  to  consider 
the  following  issues: 

1.  ACCESS  -  Are  the  data  arangement  and  data  access  methods  (index, 
table  of  contents,  etc.)  such  that  you  can  access  the  data  efficiently  and 
speedily? 

2.  PRESENTATION  -  Are  the  format  and  manner  of  presentation  in  the 
sample  such  that  they  allow  easy  use  of  the  relevant  data? 

3.  USEFULNESS  -  Is  the  type  of  perceptual  data  represented  by  this 
sample  of  value  to  the  ATD  design/development  process? 
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In  the  following  pages  we  will  ask  specific  questions  relating  to  these 
points.  Please  feel  free  to  comment  on  any  aspects  of  the  material,  whether 
we  have  asked  about  them  or  not.  Remember  also  that  ultimately  the 
engineering  community  will  be  the  user  of  this  material  and  that  it  should 
therefore  reflect  the  needs  of  that  community. 

In  order  to  allow  you  to  place  our  ICR  review  questions  in  better  context,  we 
have  included  here  a  series  of  hypothetical  "design  questions"  which  might  be 
asked  of  an  ATD  engineer.  Please  use  these  design  questions  to  help  judge  the 
ease  of  access,  the  format  and  presentation  of  the  data,  and  the  value  of  the 
data  in  solving  such  questions. 

Your  assistance  in  this  review  is  greatly  appreciated. 

Before  proceeding  further  with  your  evaluation,  please  refer  to  the  following 
portions  of  the  sample  ICR  documentation.  These  will  help  set  the  stage  for 
the  evaluation  comments  and  questions. 

Foreword 

Contents  (both  short  and  long  versions) 

Section  1.0  Introduction  to  ICR 
Section  2.0  How  to  Access  ICR 

DESIGN  QUESTIONS 

Because  you  will  be  better  able  to  assess  the  ICR  documentation  after  using 
it,  we  have  included  several  hypothetical  design  questions  below.  We  suggest 
you  use  a  few  of  these  or  similiar  design  questions  to  try  out  the  entry 
methods  provided  in  the  Users  Guide  and  then  to  assess  the  presentation  of  the 
data  in  the  Data  Base.  You  are  welcome  to  substitute  your  own  design 
questions,  but  please  remember  that  only  part  of  the  Users  Guide  and  two 
sections  of  the  Data  Base  are  completed. 

When  you  finish  on  this  page,  the  following  pages  provide  space  for  you  to 
record  you r  assessment  of  the  accessibility,  presentation  and  usefulness  of 
the  data. 
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(1)  A  simulator  is  being  designed  to  train  nap-of-the-earth  flight. 
Interaction  with  other  aircraft  will  not  be  trained  in  this  device.  A 
visual  display  will  be  placed  in  the  windscreen  area  immediately  in  front 
of  the  aircraft,  filling  about  20  degrees  on  either  side  of  the 
centerline.  Assume  that  because  of  design  constraints  only  a  small 
portion  of  the  remainder  of  the  pilot's  field  of  view  can  be  provided  with 
a  visual  display.  What  areas  have  highest  priority  for  this  display? 

(2)  What,  if  any,  are  the  detrimental  effects  of  allowing  the  raster 
lines  to  be  visible  in  an  ATD  visual  display  intended  for  training 
nap-of-the-earth  flight? 

(3)  Assume  that  a  new  algorithm  for  generating  texture  in  a  CIG  scene  has 
been  developed.  It  is  appealing  because  it  provides  a  computationally 
efficient  way  of  providing  more  edges  in  the  scene.  However,  the  texture 
is  not  perspecti vely  correct  because  it  is  generated  in  the  screen  plane, 
rather  than  in  image  space.  This  means  that  when  the  texture  is  placed  on 
a  surface  that  slopes  away  from  the  observer,  the  texture  elements  do  not 
get  smaller  with  distance.  Is  this  texture  suitable  for  a  visual  display 
to  be  used  to  train  low-level  flight? 

(4)  The  texture  in  (3)  is  composed  of  regularly  spaced  elements,  all  the 

same  size.  Is  this  better  or  worse  than  another  texture  composed  of 

irregularly  sized  elements  with  an  irregular  spacing? 

(5)  How  many  image  details  (edges,  faces,  etc.)  should  be  provided  in  a 

CIG  visual  scene  so  that  it  will  be  most  effective  in  giving  a  pilot  a 

sensation  of  actual  flight  in  a-nap-of-the-earth  trainer? 

(6)  In  ATDs,  a  display  portraying  external  visual  scenes  is  generally 
designed  so  that  the  image  is  at  a  distance  near  optical  infinity,  say 
beyond  20  to  40  feet.  What  data  support  the  additional  expense  entailed 
in  placing  the  displayed  image  at  this  distance,  rather  than  much  closer? 
For  example,  it  would  be  much  less  expensive  to  dispense  with  the  typical 
collimating  mirror  and  beamsplitter  and  simply  mount  the  CRT  display  just 
outside  the  aircraft  windows. 
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Now  that  you  have  looked  at  the  various  portions  of  the  sample  ICR  documenta¬ 
tion,  please  give  us  your  impressions.  In  addition  to  the  specific  questions 
below,  we  welcome  any  additional  comments  or  suggestions  you  may  have.  If  you 
wish,  you  may  write  consents  in  the  ICR  documentation.  (We  will  supply  you 
with  a  replacement  copy  if  you  want  one.) 

Because  your  time  is  limited,  the  questions  below  are  phrased  to  cover  the 
whole  documentation  package.  However,  multiple  responses,  each  referenced  to 
specific  portions  of  the  documentation,  are  especially  helpful. 


A.  ACCESS 

Al.  How  easy  was  it  to  locate  material  relevant  to  a  particular  topic? 


A2.  Which  methods  of  access  did  you  use? 


A3.  Which  method(s)  of  access  are  most  effective? 

B.  PRESENTATION  OF  DATA 

Bl.  How  easy  was  the  presentation  format  to  follow? 


B2.  How  easy  was  it  to  interpret  the  data  and  recommendations  presented? 


83.  Did  the  descriptions  of  experiments  contain  sufficient  detail  to  allow  you 
to  judge  the  applicability  of  the  experimental  data  to  your  specific  design 
questions? 
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B4.  Did  the  presentation  allow  you  to  see  the  extent  to  which  a  recommendation 
was  supported  by  research  data? 


! 


B5.  Are  there  additional  aspects  of  the  presentation  on  which  you  wish  to 
cornnent? 


C.  USEFULNESS 

Cl.  How  useful  a  resource  will  the  ICR  documentation  be  in  its  final  form? 
(Base  your  response  on  the  small  sample  you  have  seen,  plus  the 
indications  of  eventual  content  provided  by  the  tables  of  contents  and 
other  portions  of  the  Users  Guide.) 

_ Extremely  useful 

_ Very  useful 

_ Moderately  useful 

_ Slightly  useful 

_ Not  useful 

C2.  What  additional  ATD  design  topics  or  types  of  perception  data  particularly 
deserve  attention  in  the  ICR  study?  {The  tables  of  contents  and  other 
portions  of  the  Users  Guide  provide  an  indication  of  the  content  currently 
planned.) 


